Computational Learning Theory

Valiant introduces a precise computational model for concept learning. Concept learning is about learning to decide whether a certain data belongs to a certain concept (is this a table? Is there an elephant in the data?). The paper is the birth of PAC-learning (PAC = Probably Approximately Correct). What is a concept? A concept is simply a subset of some domain. A concept comes from a some concept class C known to the learner. So the learner is not completely in the dark. Rather, he/she knows that the concept to learn comes from a certain concept class. How does one get information about a concept? Either passively by examples. The learner is shown examples drawn according to some unknown probability distribution D. With each example comes the information, whether the example belongs to the concept or not. Or actively, by oracle calls. The learner can ask whether an specific object which he/she creates belongs to the concept or not. In later papers there is also supervised learning where a teacher provides strategically chosen examples. What does it mean to have learned a concept? It means to produce a hypothesis h from some hypothesis class H that is close to c, i.e., is able to classify unknown examples with high confidence (= small error probability). The examples on which the learner has to show its skills are drawn according to the same probability distribution D which is used in the learning phase. What does it mean to have a learning algorithm for a concept class? The algorithm should be able to learn any concept in the concept class with high probability of success (= small failure probability) after seeing a polynomial number of examples.